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Abstract 

This note is concerned with a diluted version of the perceptron 
model. We establish a replica symmetric formula at high temperature, 
which is achieved by studying the asymptotic behavior of a given spin 
magnetization. Our main task will be to identify the order parameter 
of the system. 
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1 Introduction 

A wide number of spectacular advances have occurred in the spin glasses 
theory during the last past years, and it could easily be argued that this 
topic, at least as far as the Sherrington-Kirkpatrick model is concerned, has 
reached a certain level of maturity from the mathematical point of view: 
the cavity method has been set in a clear and effective way in some 
monotonicity properties along a smart path have been discovered in j3] , and 
these elements have been combined in ^U] in order to obtain a completely 
rigorous proof of the Parisi solution [7]. 

However, there are some canonical models of mean field spin glasses for 
which the basic theory is far from being complete, and this paper proposes 
to study the high temperature behavior of one of them, namely the diluted 
perceptron model, which can be described as follows: for > 1, consider the 
configuration space S^r = { — 1, 1}'^, and for a = (ai, . . . , cttv) G S^v, define 
a Hamiltonian —Hn,m{o') by 



In this Hamiltonian, M stands for a positive integer such that M = aN 
for a given a G (0,1); u is a bounded continuous function defined on M; 
{gi,k, i ^ > 1} and {7i,fc, i > > 1} are two independent families 
of independent random variables, gi^k following a standard Gaussian law and 
7j,fc being a Bernoulli random variable with parameter which we denote 
by -B(-^). Eventually, {rjk, A; > 1} stands for an arbitrary family of numbers, 
with rjk G {0, 1}, even if the case of interest for us will be r/fc = 1 for all 
k < M. Associated to this Hamiltonian, define a random Gibbs measure Gn 
on Sat, whose density with respect to the uniform measure /xat is given by 
Z^^^j^exp {—Hpf^M{(^)), where the partition function Zj^^m is defined by 




(1) 




In the sequel, we will denote by (/) the average of a function / : S 
with respect to dGf/^, i.e. 



'N 



n 
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The measure described above is of course a generalization of the usual 
perceptron model, which has been introduced for neural computation pur- 
poses (see jS]), and whose high temperature behavior has been described in 
Chapter 3], or jS] for an approach based on convexity properties of the 
Hamiltonian. Indeed the usual perceptron model is induced by a Hamiltonian 
Hn,m on Stv given by 




where we have kept the notations introduced for equation Thus, our 
model can be seen as a real diluted version of 0, in the sense that in our 
model, each condition Y^^^j^ gi,kli,k(^i > only involves, in average, a finite 
number of spins, uniformly in N . It is worth noticing at that point that 
this last requirement fits better to the initial neural computation motivation, 
since in a one-layer perceptron, an output is generally obtained by a threshold 
function applied to a certain number of spins, that does not grow linearly 
with the size of the system. Furthermore, our coefficient 7 is arbitrarily large, 
which means that the global interaction between spins is not trivial. Another 
motivation for the study of the system induced by (H)) can be found in [2]. 
Indeed, in this latter article, a social interaction model is proposed, based on 
a Hopfield-like (or perceptron-like) diluted Hamiltonian with parameter N 
and M, where represents the number of social agents, and M the diversity 
of these agents, the number of interactions of each agent varying with the 
dilution parameter. However, in the equilibrium of the system is studied 
only when M is a fixed number. The result we will explain later on can thus 
be read as follows: as soon as the diversity M does not grow faster than 
a small proportion of A^, the capacity the social interaction system is not 
attained 

Let us turn now to a brief description of the results contained in this 
paper: in fact, we will try to get a replica symmetric formula for the system 
when M is a small proportion of A^, which amounts to identify the limit of 
\og{Zf4^M) when A^ — >• cxo, M = aA^. This will be achieved, as in the diluted 
SK model studied through the cavity method (see |2] for a study based on 
monotonicity methods), once the limiting law for the magnetization ((Xj) is 
obtained. This will thus be our first aim, and in order to obtain that result, 
we will try to adapt the method designed in Chapter 7]. However, in 



3 



our case, the identification of tlie limiting law for (cTj) will be done through 
an intricate fixed point argument, involving a map T : P — P (where P 
stands for the set of probability measures on [—1,1]), which in turn involves 
a kind of 'P(A)®'^*^'^) measure, for two independent Poisson measures P(A) 
and V{fi). For sake of readability, we will give the details of (almost) all 
the computations we will need in order to establish our replica symmetric 
formula, but it should be mentioned at that point that our main contribution, 
with respect to P, Chapter 7] , is that construction of the invariant measure. 

More specifically, our paper is divided as follows: 

• At Section 121 we will establish a decorrelation result for two arbitrary 
spins. Namely, setting Uoo = ll^^lloo, for aUoo small enough, we will 
show that 



K 



for a constant K > 0. 



At Sectional we will study the asymptotic behavior of the magnetiza- 
tion of m spins, where m is an arbitrary integer. Here again, if alloo is 
small enough, we will see that 



E 



i<m 



< 



N 



where Zi, . . . , Zm is a family of i.i.d random variable, with law fia,'y, and 
the fixed point of the map T alluded to above, whose precise 
description will be given at the beginning of Sectional 

Eventually, at Section we obtain the replica symmetric formula for 
our model: set 



G(7) 



1<P 



a 



log [ 2_^exp(-7)— E 

p=0 



V„ 



where means integration with respect to the product measure i' 
on { — 1, 1}^ such that J aidv = Xj. Let F : [0, 1] M"*" be defined by 



4 



F(0) = log2 — aM(0) and -^'(7) = G'(7). Then, if aUoo is small enough, 
we will get that 



-E [log(Z^,M)] - F{-f) 



K 



for a strictly positive constant K. 



All these results will be described in greater detail in the corresponding 
sections. 



2 Spin correlations 

As in [9, Chapter 7], the first step towards a replica symmetric formula will 
be to establish a decorrelation result for two arbitrary spins in the system. 
However, a much more general property holds true, and we will turn now to 
its description: for j < N, let Tj be the transformation of that, for a 
configuration (a\ . . . ,<t") in S^, exchanges the j-th coordinates of and 
cr^. More specifically, let / : M, with n > 2, and let us write, for 

J<N, 

t =/ (a^.c,a^.,a^.c,a^.,...,a^c,aj, 

where, for Z = 1, . . . , n, cr^c = ((t{ . . . , cr\^i, o'j+i; • • • ; cjv)- Then define / o Tj 
by 

/ o T, . . . , a") = / (a]., a|; a% a); . . . ; a^., a]) . (3) 
For J < — 1, we will call Uj the equivalent transformation on 

Definition 2.1 H/'e say i/ia^ Property P{N, 70, -B) satisfied if the following 
requirement is true: let f and f be two functions on depending on m 
coordinates, such that / > 0, /' o T/v = — /', and there exists Q > such 
that l/'l < Qf; then «/7 < 70 we have 



E 



(/') 



^ mQB 



for any Hamiltonian of the form (Qp, uniformly in rj. 

Set now Uoo = H^^Hoo- With Definition 12.11 in hand, one of the purposes of 
this section is to prove the following Theorem. 
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Theorem 2.2 Let 70 be a positive number, and Uoo be small enough, so that 

a^l e'^- e'^^o^^^"--^) (3 + 270 + a(7o' + 7o')e'^-) < 1- (4) 

Then there exists a number Bq{jq,Uoo) such that if •y < 70, the property 
P{N, 7o, -Bo) holds true for each N > 1. 

In the previous theorem, notice that the value of 70 has been picked arbitrar- 
ily. Then we have to choose Uoo, which also contains implicitly the tempera- 
ture parameter, accordingly. Let us also mention that the spin decorrelation 
follows easily from the last result: 

Corollary 2.3 Assuming ^ there exists K > such that, for all 7 < 70, 

E|(aia2)-(ai)(a2)|<^. 



Proof: It is an easy consequence of property P(A^, 70, -Bo) applied to n = 2, 
/ = land f'ia\a')=al{al-al). 

□ 

We will prepare now the ground for the proof of Theorem 12.21 which will 
be based on an induction argument over N. A first step in this direction will 
be to state the cavity formula for our model: for cr = (cxi, . . . , ctat) G Eat, we 
write 

P = • • • ; PN-i) = (O"!, • • • , (^N-l) e T^N-l- 

Then the Hamiltonian (^Q) can be decomposed into 

-HN,Micr) = ^ Vk7N,kU I ^ gi,kli,k<yi + 9N,kO-N I - H^-i^kAp)^ 

k<M \i<N-l / 

with 

-Hn-i,kAp) = ^Vk^[ XI 9i,kli,k(Ti j , and r/" = 7]kil--fN,k)- (5) 

k<M \i<N-l J 

Note that in H]^_^ j^^j, the coefficients 77^ = rik{l — 'yN,k) are not deterministic, 
and hence H^_^ is not really of the same kind as Hj^^m- However, this 
problem can be solved by conditioning on {'jN,k,k < M}. Then, given the 
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randomness contained in the '^N,ki the expression H^_^ j^j{p) is a Hamiltonian 
of a (A^ — l)-spin system with 'ji^k ~ B(^—^), where 7" = 7"^^^ and so 
7" < 7 < 7o- 

Thus, given a function / : — > M, we easily get the following decom- 
position of the mean value of / with respect to Gfj^: 

If) = <^"^'^>- (6) 

with 

^ = exp I rik-fN,kU I gt,kli,k(Tl + gN,k(yN I 1 , (7) 

\l<n k<M \i<N-l J J 

and with (/)_ defined, for a given / : M, by 

E(pi,...,pn)GE^_, fip\ • • • 'P") exp (- E/<n ^JV-1,m(p')) 



(/)- 



Notice also that in expression ©, Av stands for the average with respect to 
the last component of the system, namely if / = f{p^,a\f,...,p^,a%), then 



Let us introduce now a little more notation: in the sequel we will have to 
take expectations for a fixed value of ^ given at ((Zj). Let us denote thus by 
E^^ the expectation given 7Ar,fc, k < M, and define 

E_,^^[ ■ ] = E^^ [ ■ I gN,k, gi,k, li,k, i<N-l, ke D^^^ ] , (8) 

where is given by 

Df,\, = {k<M; ^N,k = 1}. 

One has to be careful about the way all these conditioning are performed, 
but it is worth observing that the set D^-^ is not too large: indeed, it is 
obvious that, setting \A\ for the size of a set A, we have 

k<M 
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and thus 



E\D 



M I 

na\ 



M— 

N 



07. 



Let us go on now with the first step of the induction procedure for the 
proof of Theorem l2.2l in P(A^, 70, B) we can assume without loss of generahty 
that / and /' depend on the coordinates 1, . . . ,m — l,iV. Moreover, since 
\n\ < Qfi. we have 



and hence 



|(Av/'0-|<(Av|/'e|)_<(gAv/0^ 



(Av/0- 

We now define the following two events: 



<Q. 



(10) 



{3p<m-l,keD^,; 7^ = 1} 

{3p <m-l,k < M; 7^,^ = 7N,k = 1}, 

{3j <N-l, h, k2 G D^Jy, 7,- fc, = 7,- fc, = 1} 
{3j < N ~ l,ki,k2 < M; -fj^k^ = IjM = Inm 



INM = !}■ 



These two events can be considered as exceptional. Indeed, it is readily 
checked that 



m)<a^(m-l). 



N -1 



Thus, if = l^i U r22, we get 



a'y'^{m — 1) + a'^^'^ 
N ' 



and using this fact together with (jlOj) . we have 



E 



(/') 



= E 
= E 

< Q 



(Av/'O^ 



(Av/e). 

(Av/'O- 



(Av/0- 
07^ (m — 1) + 0^7'^ 
N 





(Av/'O- ^ 


(Av/0- , 




(AvfO- 


(Av/O- 



Consequently, in order to prove Theorem 12.21 we only need to bound accu- 
rately the expectation of the right-hand side of by means of the induction 
hypothesis. To this purpose, we will introduce some new notations and go 
through a series of lemmas: set 

Ji = {j < N; jj^k = 1 for some G D^^^} - {A^}, 

and observe that, when fli does not occur, 

Jin{l,...,m-1} = 0. 

Denote |Ji| = card(Ji) and write an enumeration of Ji as follows: Ji = 
{ji, ■ ■ ■ ,j\Ji\}- 

Lemma 2.4 Let Uj be the transformation defined at and f : M 
such that f o T/v = — /'. When Vti does not occur, we have 

(Av/'o ° n = -Av/'e. 

jeJi 

Proof: The proof of this lemma can be done following the steps of p', Lemma 
7.2.4], and we include it here for sake of readability. Set T = HjeJi ^j- Since 
/' depends only on the coordinates {1, . . . , m — 1, iV} and this set is disjoint 
from Ji, we have /' o T = /'. Moreover, 

// rji rji £1 rri pf 

oT oTn = f oTn = - f ■ 

On the other hand, ^ only depends on Ji U {N} and using 

e(a^a^...,a")=e(o^^a^...,a"), 

we obtain 

^ oT oTn = C 

Hence 

(/'OoToT^ = -/'e, 

and, since = Id, we get 

{fO°T = -{fOoT^. (12) 
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Eventually, 



(13) 
(14) 



The proof is now easily concluded by plugging (fTSj) and (fT^ into (fT^ . 

Let us now go on with the proof of Theorem 12.21 thanks to Lemma |2I 
when Qi does not occur, we can write 



□ 



Av/'e - (Av/'O o J] [/^.^ 

s<|Ji| 



l<S<|Jl| 



with 



/. = (Av/'O o H U,, - (Av/'O o n U,, 



Ks-l 



Ks 



Notice that Uj = Id, and that fs enjoys the same kind of antisymmetric 
property as /', since o Uj^ = -f^. 

Define Ri = l-Djvil- Then, recalling relation Q, we have 

-Rl = l-^TV^ll = lN,k, 
k<M 

and let us enumerate as ki, . . . , k^^ the values k < M such that '~fN,k = 1- 
We also define ll, . . . , as follows: 

4' = {j<N-l; 7,- fc„ = 1}, for v < 

and observe that we trivially have 

Ji = U ll (16) 

v<Ri 

Moreover, when ^2 does not occur, we have 
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Then, on Q'^, we get 



|Ji| = Card(Ji) = ^ 



(17) 



t;<_Ri 



Furthermore, it is easily checked that, for each v, and conditionally on the 
7Ar,fc, the quantity |/^| is a binomial random variable with parameters N — 1 
and ^, which we denote by Bin(A^ — 1, -^). 

With all these notations in mind, our next step will be to bound fs in 
function of /, in order to get a similar condition to that of Definition 12.11 

Lemma 2.5 Recall that Uoo = ||w||oo- Then, on Q'^, for js G I^, we have 
where 



Proof: Let us decompose ^ as ,^ = ^' with 



\3</<n k<M \i<N-l / / 

C = exp I ^ ^ VklN,kU I ^ 9i,kli,kcr\ + gN,k(yN I I • (18) 

\ l<2 k<M \j<Af-l / / 



Thus 



\ l<2 v<Ri 

> e'exp(-2f/ooi?i). 



u 



9i,kvli,k^(^\ + 9N,kv(^N 1 
i<N-l J 



and hence 



Av/e> (Av/Oexp(-2[/ooi?i) 



(19) 
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On the other hand, since /' only depends on {1, . . . ,m — 1, A^}, we have 
/' o Tj^ = f for any / < | Ji| on Q!^^ which yields 

/. = ( Av/'o o n u,, - ( Av/'o o n u,, 

l<s-l 1<S 

= Av((/'Oo n T,,-{n)ol[T,] 

\ 1<S-1 1<S J 

= Av(/'ko n T,,-^ol[T,]Y (20) 

V \ Ks-1 l<s J J 

where we have used the fact that Ji can be written as Ji = {ji, • • • , 
Moreover, for any /, by construction of we have ^' o Tj^ = Thus, 



Set now 



r = sup 



Ks 



1<S~1 



Ks 



(21) 



Co n - roji^. 



Ks-l 



Ks 



Then, from ()20p and ()21|) . and invoking the fact that |/'| < Qf, we get 

|/,| < TAvd/'IO < grAv/e'. (22) 
We now bound F: recall that ^" is defined by (jl8j) . and thus 

e = n 

v<Ri 

with 

Cv = expiY^ r]k,u gi,k^li,kA + 9N,k,(y^N I 

\«<2 \j<7V-l /. 

Recall now that we have assumed that js G Therefore, we have js ^ II if 
w 7^ f , according to the fact that fl = on il"^. Hence 



Cv ° 
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and 

On the other hand, since |e^ — e^| < |x — ?/|e° for |?/| < a, we obtain 
and we also have the trivial bound 

Thus, plugging (j211) and (j2Sl) into ((221), we get 

r < AU^e'^^°^^\ 



(23) 



(24) 



(25) 



Combining this bound with (jl9j) and (j22p . the proof is now easily completed. 

□ 

We are now ready to start the induction procedure on P(A^, 70, -B), which 
will use the following elementary lemma (whose proof is left to the reader). 

Lemma 2.6 Let R he a random variable following the Bin(M, -^) distribu- 
tion, and X be a positive number. Then 



Let us proceed now with the main step of the induction: 



(26) 
(27) 



Proposition 2.7 Assume that P (N — 1,'jo, B) holds for N > 2 and 7 < 79. 
Consider f and f as in Definition \2.1\ Then 



E 



(/') 



<^(a7' + aV + 4i?T(a,7,f/oo)) 



(28) 



where 



T(a, 7, f/00) = f/oo«7' e'^- e"^(^'"°°-i) (3 + 27 + a{^' + 7=^)e^^-) 
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Proof: Using (fTT|) and (fT^ . we have 



E 



(/') 



Q;7^(m - 1) + 0^7'' 



< Q 



\{fs)- 



s<\Ji\ 



(Av/0- 



However, on fi*^, the functions fs and Av/^ depend on m — 1 + | Ji| coordi- 
nates. Since 7~ < 7 and m — 1 + | Ji| < m(l + | Ji|), the definition of the 
expectation E_^^j^, the property P(A^ — 1, 70, B), (fTTjl and Lemma imply 



E 



s<\Ji\ 



(Av/0- 



E 



If^^ 5^ E_, 

s<\Ji\ 



In 



\{fs)- 



(Av/0- 



< E 



s<|Jl| 

m 



(m-l + |Ji|)Eg 
iV- 1 



< 4^^figt/oo E Ji|(l + I Jil) e^^-^i] 



< 8-BQU^ E [lnc| Ji|(l + I Jil) e^^-^^] . 
Recall that, according to (fT^ . we have 

?;<Hi 

and that the quantity -Ri is a Bin(M, -^) random variable. Thus 



E[lnc|Ji| e^^^] 



E <! e^^^ E 



Ri 

Ri 
Ri 



E <! e^^i E 



Ri 



< (7 + 7')E[(i?i + i??)e^^^^] 



(29) 



The proof of this proposition is now easily concluded by applying the previous 
bounds, together with Lemma IT^ to the quantity 

E[l^.|Ji|(l + |Ji|)e^^-^i]. 
14 



□ 



We can turn now to the main aim of tliis Section: 

Proof of Theorem 12. 2t Tlie result is now an immediate consequence of 
(jlj) and Proposition \2.7\ applied to 

B = Bq = , 

1 — e 

where e satisfies 

AU^ «7o e"""^ e°^°('='''°°-i) (3 + 270 + a{^l + iDe^""-) <e<l. 

□ 



Before closing this Section, we will give an easy consequence of Theorem 
12.21 we will see that, as N grows to 00, the Gibbs measure Gn taken on a 
finite number of spins looks like a product measure. To this purpose, let us 
denote by (■), the average with respect to the product measure v on S^v-i 
such that 

V^<iV-l, j a.,dv{p) = {ai).. 
Equivalently, for a function / on Sjv_i, we can write 

(/). = (/>!,..., 

where al is the i-th coordinate of the i-th replica p*. Recall also that, for 
V < Ri, II has been defined as 

ll = {i<N- 1; 7,,fc„ = 1}. 

We now introduce the enumeration . . . of this set. Furthermore, 

given the randomness contained in the 77v,fc, the law of |J^| is a Bin(A^ — 1, -^). 

Proposition 2.8 Assume ^ and 7 < 70, and consider 

e = exp ^ 9ii,k,(^ii + gN,k.(^N 

\v<Ri \p<\il\ 
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Then, when Q does not occur, we have 

(AvcrAf0)_ (Avo-Are). 



E_ 



,7iv 



(Av0)_ (Av0). 



<2i?o(|Ji|-l) ^^(e^^^ 



where the conditional expectation E_^^j^ has been defined at 

Remark 2.9 The quantity appears naturally in the decomposition of the 
Hamiltonian Hj^^m- Indeed, on we have 

-HN,Mi^) = ^Vkul ^ 9i,k'yi,kcri I 

k<M \i<N J 

= ^ VkU { ^ 9i,kli,kO-i j + ^ VkU { ^ gi,kli,kO-i + 9N,k' 

kiDM_^ \i<N-l J kGDfi^ \i<N-l 

= ^ VkU i 9i,kli,k(^i j + Vk^u 9i-,k^cri- + 9N,k^(TN 

k^Df,\^ \i<N-l J v<Ri \p<\Ii\ 

Observe also that ^ defined by ^ evaluated for n = 1 gives ^ = Q. 

Proof of Proposition l2T8l The proof is similar to Proposition 7.2.7 in [0], 

and we include it here for sake of completeness: On Q'^, since the sets are 
disjoint, the values i^, for any v and p, are different and we can write 

U 4^ = Ji = {ji,...j|,,|}. 

v<Ri 

Set 

/' = • • • , ^iJ - Ava^0, f = fial...,ajj^ Av0. 

Let us also define, for 2 < / < | Ji|, 
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and /,■; in a similar way. Then 



E_ 



,7iv 



(Avo-jv6)_ (Avo-atG), 



E 



(Av0)_ 

(4)- 



,7jv 



(Ave). 



2<;<|Ji| 



E_ 



7JV 



(/j|Ji|) 



s E 

2</<|Ji 



,7]V 



+ E_ 



,1N 



ifn-i)- ifn)- 
{fn)-{fn-i-fn) 



(30) 



Let us concentrate now on the first term of the right-hand side of (jHUI) . since 
the other term can be bounded similarly: observe that, for 2 < I < |Ji|, we 
have 

f'.^ = f'.^_^A, with e-2^°° < A < 6^^°°. 

Furthermore, it is easily seen that — fj^ enjoys the antisymmetric prop- 
erty assumed in Definition I2.ll Thus, applying P(A^ — l,7o, Bq), we get 



E_ 



,1N 



{fj,-l fj,)- 



< 



Boi\Ji\ + ll 
N-1 ^ 



-1), 



which ends the proof. 



□ 



3 Study of the magnetization 

For the non-diluted perceptron model, in the high temperature regime, the 
asymptotic behavior of the magnetization can be summarized easily: indeed, 
it has been shown in ^ that (ai) converges in to a random variable of 
the form tanh.'^ (zy/r), where r is a solution to a deterministic equation, and 
z ~ N{0, 1). Our goal in this section is to analyze the same problem for the 
diluted perceptron model. However, in the current situation, the limiting law 
is a more complicated object, and in order to present our asymptotic result, 
we will go through a series of notations and preliminary lemmas. 

Let P be the set of probability measures on [—1,1]. We start by con- 
structing a map T : P — >■ P in the following way: for any integer ^ > 1, 
let (ti, . . . , Tg) be 9 arbitrary integers. Then, for /c = 1, . . . , 6*, let be the 
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cumulative sum of the r^; that is, to — and tk = X]A;<fc '^k ^'^^ k > 1. Let also 
{cji^k, hk > 1} and {g^, A; > 1} be two independent families of independent 
standard Gaussian random variables. Define then a random variable ^o^r by 

= exp ^ 14 I ^ Qi^k (^tk-i+i + 5fe £ 1 ■ (31) 
fe=i \i=i / 

Whenever ^ = 0, set also = 1, which is equivalent to the convention 
Y^k=i '^k — ^ for any real sequence {wk] A; > 0}. 

Consider now x — (xi, . . . , x^e ,^^) with \xi\ < 1 and a function 

/ : {-1,1}^^=!^'= ^M. 

We denote by (/)a; the average of / with respect to the product measure v on 
{— 1, l}Sfc=iT-fc such that r aidu(5) — Xi, where 5 — ((Ji, . . . , a^^e ). Using 
this notation, when ^ > 1, we define Tq^t : P — > P such that, for // e P, 
Te^ri/J') is the law of the random variable 



(32) 



where X = (Xi, . . . , X^9 ) is a sequence of i.i.d. random variables of law 

2-^ fc=l 

Ijl independent of the randomness in ^g^T- and Av denotes the average over 
e — ±1. When ^ = we define Tq^t{ij) as the Dirac measure at point 0. 
Eventually, we can define the map T : P ^ P by 

^(/^) = X] X] ^(^' '^1' • • • ' '^e) Te,r{lj)^ (33) 
e>o Ti,...,Te>o 

with 

«(e,ri, . . . ,r,) = e-"^ ^ e"^^ ^. -, (34) 

u\ Ti! • • • T0\ 

and where the coefficients a, 7 are the parameters of our perceptron model. 
We will see that the asymptotic law /i of the magnetization (ai) will satisfy 
the relation /i = T{fi). Hence, a first natural aim of this section is to prove 
that the equation = T(/x) admits a unique solution: 
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Theorem 3.1 Assume 



2Uoo e^^- < 2' (35) 

Then there exists a unique probability distribution /x on [—1,1] such that 
/i = T(^). 

Remark 3.2 Notice that ^ implies 

In order to settle the fixed point argument for the proof of Theorem I3.H 
we wiU need a metric on P, and in fact it wiU be suitable for computational 
purposes to choose the Monge-Kantorovich transportation-cost distance for 
the compact metric space ([—1, 1], | • |): for two probabilities /xi and /i2 on 
[—1,1], the distance between fj,i and fj,2 will be defined as 

rf(/ii,/i2) = infE|Xi -X2I, (36) 

where this infimum is taken over all the pairs {Xi,X2) of random variables 
such that the law of Xj is nj, j = 1,2. This definition is equivalent to say 
that 

(i(/ii,/i2) = inf J d{xi,X2)d({xi,X2), with d{xi,X2) = \x2 - Xi\, 

where this infimum is now taken over all probabilities ( on [—1, 1]^ with 
marginals /ii and fi2 (see Section 7.3 in [9^ for more information about 
transportation-cost distances). Finally, throughout this section, we also use 
a local definition of distance between two probabilities, with respect to an 
event fl: 

rfc(/ii,/i2) = infE|(Xi-X2)lc|, (37) 
where this infimum is as in (jHE))- 

Proof of Theorem 13.11 Assume that 6 > 1 and > 1 for some k = 
1, ■ ■ ■ ,6. Then, using similar arguments to Lemma 7.3.5 in [5] we can prove, 
for 1 < z < Ylk=i '^k, that 



dxi {A\ie^r)i 



<2f/ooe2^°°, (38) 
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with ^^^). Then if y = . . . ^t-*.)' bound (|HH|) 

imphes that 



< 2U^ e^^- J] J] - (39) 

fc=l i=l 

Remark that if 9 = ot 6 ^ but = for any /c = 1, . . . ,6, then the 
left-hand side of (jH^ is zero. 



Let now (X, Y) be a pair of random variables such that the laws of X and 
Y are /ii and /i2, respectively (/ii and /i2 are independent of the randomness in 
(.e^r)- Consider independent copies {Xi,Yi)._^-^e of this couple of random 
variables. Then, if X = {Xi)-^^e and Y = {Yi)-^^e , we have that 

{Ave^e,r)x (d) , . (Av£^0,^)y (d) 

Hence, applying (jHTIjl for x = X and y = Y and taking first expectation and 
then infimum over the choice of (X, Y), we obtain 

e 

ci(Te,,(/ii), Te,,(/i2)) < e^^- ci(/ii, /ia) r^. (40) 

k=l 

Eventually, recall (see [HI Lemma 7.3.2]) that for a given sequence {c„; n > 1} 
of positive numbers such that Ylin>i = 1, and two sequences n > 1} 

of elements of P, we have 

d I ^ C„/in, ^ C„Z/„ I < ^ C„ (i (/in, ^^n) • (41) 
\n>l n>l / n>l 

Applying this elementary result to cg^r = ti, • • • , tq), /ig.r = Tg^rifJ'i) and 
t'e.T = T0_^(/i2), we get 

d{T{fii), T(/i2)) < /«(6', Ti, . . . , re) ci(T0,^(/ii), Te,^(/i2)) 

6»>0 Ti,...,Te>0 

<2f/ooe2^°°^ ^ ^K(e,ri,...,re)rfcrf(/ii,/i2) 

6»>0 ri,...,re>0 k=l 



2f/oo 6^^°° C?(/ii,/i2), 



2f/oo e^^- I 5^ e-"^ ^ ) d{f,,,f^,) 

,9>0 



20 



where we have used the fact that the mean of a Poisson random variable 
with parameter p is p. Then, under assumption (|35|). T is a contraction and 
there exists a unique probabihty distribution such that p = T{p). 

□ 

Notice that the solution to the equation p = T{p) depends on the pa- 
rameters a and 7. Furthermore, in the sequel, we will need some continuity 
properties for the application (a, 7) 1— >■ Pa,'y Thus, we will set p = pa,'y 
when we want to stress the dependence on the parameters a and 7, and the 
following holds true: 

Lemma 3.3 If {a,'j) and (a', 7') satisfy i35]) . then 



I7 - 7'|aVe'^"^'' + |«7 - aV|e'"^""'^'' 



Proof: Since pa^-y = Ta^^^pa^-y) and pa',Y = Ta'^Y{pa',-y'), using the triangular 
inequality and Theorem 13. II we have 

'^(/^Q:,7; ) ^ '^(-^a,7 (^''0,7) 5 -^a,7 (A''a',7' ) ) ~l~ '^(-^a,7 (A''a',7' ) ' '^ck',7' (/^C('i7' ) ) 



So 



d{Pa,'y, fJ-a^y) < 2(i(TQ,^-y (/iQ,',y ) , T^/^y (/iQ,',y ) ) 

and we only need to deal with c?(Ta,-y(/iQ,',y), T^,/ y (/i^/ y)). However, Lemma 
7.3.3 in jH] implies that 

'^(^a,7 (/^a',7' ) ? ^a' ,7' (/^a',7' ) ) 

<2^ 5Z \f^a,-y{d,Ti,...,Te) - Ka',y{9,n,...,Te)\ 

e>0 Ti,...,Tg>0 

<2{V, + V2), 
with K defined in (EH) and 



Vo 



0>O ri,...,Te>0 

E E ' 

e>0 -ri,...,re>0 



-07 ^ 



^! ri!---re! 



e-"^(«7)^-e-"'^'(aY)' 



-07 



^! Ti\---Te\ 
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Now, following the arguments of (7.53) in we get 



^1 = E^h""'(«^) 



'°'^'(aV) 



6»>0 



/ / 1 |q:7 — a 7 



V2 < |7-7'|el^"^'l^0e-°'^' 



[a'i) 



0>O 



e\ 



< |a7 — a 7 |e 



/ /I /I I7— 7'! 

a 7 |7 — 7 |e" ' 



which ends the proof of this lemma. 



□ 



From now on, we will specialize our Hamiltonian to the case of interest 
for us: 

Hypothesis 3.4 The parameters rjk, k = 1, . . . , M in the Hamiltonian (OP 
are all equal to one. 

This assumption being made, we can now turn to the main result of the 
section: 

Theorem 3.5 Let 70 be a positive number such that 

4t/oo «7o e"^°('='"°°-i) (3 + 270 + «(7o' + 7o')e^^-) < 1, (42) 

and assume that there exists a positive number Cq satisfying 

Coa 7o^ Uoo e'^°° < 1. (43) 

Then for any 7 < 70, given any integer m, we can find i.i.d. random variables 
zi, . . . , Zm with law fia,^ such that 



E 



< 



N 



(44) 



for a constant K > independent of m. 



Remark 3.6 The two conditions in the above theorem are met when the 
following hypothesis is satisfied: there exists L > such that 

L [/oo « 7o exp {8f/oo + «7o (e'^°° - l) } < 1. 
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As in the case of Theorem 12.21 the proof of Theorem 13.51 will require 
the introduction of some notations and preliminary Lemmas. Let us first 
recast relation ()44|) in a suitable way for an induction procedure: consider 
the metric space [—1,1]"^, equipped with the distance given by 



diiXi)i<m, {yi)i<m) = ^ 



•^i Hi I 



We also denote by d the transportation-cost distance on the space of proba- 
bility measures on [—1, 1]*", defined as in ()36|) . Define now 

D(iV,M,m,7o) = sup d {C{{a^), , (45) 

7<70 

where C{X) stands for the law of the random variable X. Then the statement 
of Theorem 13.51 is equivalent to say that, under Hypothesis (flHj) . we have 

D(N,M,m,jo) < 

for any fixed integer m > 1. 

It will also be useful to introduce a cavity formula for m spins, which 
we proceed to do now: generalizing some aspects of the previous section, we 
consider, for p G {1, . . . , m}, the random sets 

= {k<M; iN^r>+i,k = 1}, 

and 

m 

l?m _ I I T)^ 
^ N,M — \J ^N,p ■ 

p=l 

We also define the following two rare events: 

^1 = {3k < M, pi,p2 < m; jN~pi+i,k = jN-p2+i,k = 1}, 
^2 = {3i< N ~ m, ki, k2 E FJ^j^f, 7^,/,^ = 7^,^^ = 1}, 



satisfying 



99 949 
P(J^i)<^^ and P{n2)<—^. (46) 
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Then, the following properties hold true: first, for a fixed k, if (l^ is realized, 
we have 

Card{p < m, -fN-p+i,k = 1} < 1. 
Moreover, still on fll, for pi ^ P2, 



and hence. 



p=l k<M p<m 

Actually, notice that we always have 

m 
p=l 

Let us introduce now an enumeration of -F^m- 
and for any v < Rm set 

C = 0'<^-^;m = i}- 

Then, on VL^i we get 

= if^^i7^^2, (47) 

and we can also write 

Jm^ U C= U {^'^^-^^ ^^>^ = 1>- 

V<Rm V<Rm 



Let us separate now the m last spins in the Hamiltonian Hj^^m'- if is 
realized, for p = (ui, . . . , aN-m), we have the following decomposition: 



-Hn,m{(t) = -//^_^^J^^(p) +logC, 
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with 



kaiF^j^jY \i<N-m / 

^ = exp X] ^ I X] gi,kii,k<yi + gN-p+i,kO-N- 



(48) 



.P=^ keD'j^f ^ \i<N-m 



Observe that, in the last formula, H^_^ ^.j{p) is not exactly the Hamiltonian 
of a (A^ — m)-spin system changing 7 into 7", because the set -F^^^ not 
deterministic. But this problem will be solved again by conditioning upon 
the random variables {7Ar_p+i^fc, p = 1, . . . ,m, k < M}. For the moment, 
let us just mention that the m cavity formula will be the following: given / 
on Sat, we have 

where (■)_ is the average with respect to -f^^.^jvf average 
with respect to last m spins. Moreover, in the last formula, we have kept the 
notation ^ from Section |21 which hopefully will not lead to any confusion. 
Eventually, we denote by Cq the law of a random variable conditioned by 
{7Ar_p+i_A:, p = I, . . . ,m, k < M} , and by E^^^^ the associated conditional 
expectation. 

We can start now stating and proving the lemmas and propositions that 
will lead to the proof of Theorem 13.51 Recall that given x = {xi, . . . , xj\f_m), 
\xi\ < 1, and a function / on T,j^_m, {f)x nieans the average of / with 
respect to the product measure z/ on S7v-m such that / aidu^p) = Xi, for 
< i < N — m. Recall also that 7" = 7 ^"™ . Then, as a direct consequence 
of the definition of the operator t-, we have the following result: 

Lemma 3.7 Let X = (Xi, . . . ,Xjv_m) be an independent sequence of ran- 
dom variables, where the law of each Xi is fia,-/- ■ Set 

Then, on Qf^ = (Qi U we have 

£0(^1 • • ■,UJm) = T|£,Mj^(|^m|^fcg£,M^)(/i„^^-) (g) ■ • • (g)T|£,M^^|^m|_fcg£,M^)(/Xc.,7- 



^P = TTTZTl ' p=l, ...,m. 
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We will now try to relate the random variables Wp with the magnetization 
of the m last spins. A first step in that direction is the following lemma where 
we use the random value of the parameter a~ associated to the Hamiltonian 
of a [N — m)-spin system. 

Lemma 3.8 On Cl'^, set 
where, for p = 1, . . . ,m, 

^ . , ^ _ M - Rm 

Then, on Q'^, we have 

r ^ oTT 2C/oc \Rm-ma\ J \Rm - ma\\ ^.jm, 
< 2Uoo e 7o — exp <^ 70 — > 2^ 14 I- 

^ ^ k=l 



Proof: Using (jlUI) we obtain 



The proof of this lemma is then easily finished thanks to Lemma 13.31 and 
taking the following equality into account: 



7 |a — a I = 7- 



\Rm — ma\ 



N 

□ 



Notice that we have introduced the random variables idp for the following 
reason: given the randomness contained in the {77v-p+i,fc, p = I, . . . ,m, k < 
M}, Wp can be interpreted as 



^P= TTTTT^ ' p = l,...,m, 



where X = (Xi, . . . ,X^_m) is an independent sequence of random variables 
with law iia-,-r- 
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Lemma 3.9 Consider Z = (((Ti)_, . . . , (cr^v-m)-); ond denote 

(Avo-7v_p+iOz 



p = 1, . . . ,m. 



Then, on Q'^, 



d (>Co(Mi, . . . , Urn), ^o{Wl, Wm)) 

<4D(iV-m,|(F-^,r|,|J^|,7o) [/oo e^^- 
where the quantity D has been defined at relation 



Proof: As in (I38|) we can obtain, for any i < N — m, 



dxi (Av^)a; 
But in fact, these derivatives are vanishing, unless 



(50) 



7 T"^ 



U 



for some p = 1, . . . ,m. Indeed, on Q'^, from ()47|1 . we have 

Then, for a given p G {1, • • • ,Tn}, we can decompose ^ into ^ 
with 



^Af,p = exp 



' 5'i,fc7i,A:0"j + fi'Af-p+l,fcO"Af-p+l I 

keDfl^^ \i<N-m / 

^N,p £ ^N,p}^'^N-p+l) , 

^N,p e Jm\lN,p}^(^N-p+Up <m,p^p}) . 



Then 



(AvcrAr_p+i^)a; _ {Ay(Tn-p+iCn,p)x (Av^Ar^p)a. _ {AYaN-p+l^N,p)3 



{A-V^N,p)x {AvC,N,p)x 
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(Av^^,p), 



and clearly the derivative ^ is zero when i does not belong to I'^p, for any 
p e {1, . . . ,m}. 

Now, invoking inequality (|50|) . we get 



E 

p=i 



p=i i(^r 



2Uo. 



Then, the definition of E^^^ ^ and (j45j) easily yield 



E, 



7iv,7 



J2 E |^^-(^^)-|| <2DiN-m,\{FJ^^,,)%\J^\no), 



which ends the proof. 

Set now, for 1 < p < m. 



{A\aN-p+iO- 



□ 



(51) 



Then Up is closer to the real magnetization in the sense that Up = (o"Ar_p+i) 
on Cl^, and the following Lemma claims that the distance between Up and Up 
vanishes as — > cx). 

Lemma 3.10 For 1 < p < m, let Up be defined by i51]) . Then, on Q'^, we 
have 



d {jCo{Ui, Urn), ^q{Ui, • • • , Mm)) < 25o 



I / P - 1 



N -m+1 ' 

where the constant Bq has been defined in the previous section. 



-1), 



Proof: The computations can be leaded here almost like in the proof of 
Proposition I2.8| and the details are left to the reader. 

□ 

We will now identify the law of the Up in terms of laws of the type T{fia,'y- ) '■ 
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Lemma 3.11 Recall that d^c has been defined by relation (|J7p . Then, for 
m > 1, set 

Sm = d^c (c{ui, . . . y^y^g ((6i,t;i), . . . , {bm,Vm)) 

(b) (v) 

where we have used the following conventions: for j < m, Vj is a multi-index 
of the form Vj = (f^, . . . , f^.); the first summation ^■^ O'wer bj > 0, for 

j = 1, . . . , m; the second one ^■^ O'wer v{, . . . ,vl, > 0, for j = 1, . . . , m; 

and a{{bi,Vi), . . . , {bm,Vm)) is defined by 

a . . . , {bm,vm)) = P {\D^i,\ = b„ (l/fU e D^l^) = vj,\/j < m) . 

Then, under the conditions of Lemma \3.1(A we have 

Sm < ci{N, m), 

with 

ci{N,m) = 4 f/oo e'^- E(Z}(iV-m,|(F;^jv,)%|J^|,7o)) 

+2f/„oe2^- ^E(|/?^-ma| | J„| exp - ma| }) . 



Proof: This result is easily obtained by combining Lemmas 13.71 13.81 13. 9| 
I3.1(JI and taking expectations. 

□ 



With Lemma f3. Ill in hand, we can see that the remaining task left to us 
is mainly to compare the coefficients a((6i,t;i), . . . , (6m,i^m)) with the coeffi- 
cients K,a,-y-{bj,Vj). This is done in the following lemma. 

Lemma 3.12 With the conventions of Lemma \3.11l we have 



EE 

(b) (v) 



i=i 



mLni'-y) 
< (52) 



29 



N 



Proof: In fact, it is easily seen that we only need to prove that 

\a{b,v) - Ka,^-{b,v)\ < 

t,v>0 

with V = (f 1, . . . ,Vb)- However, notice that 

Vl 

and recall that 



^ ^ 1=1 



n) V iV 



Vi\ ■ ■■Vb\ 



Then 



|a(6, v) - Ka^-y- {h,v) \ < A + B, 

b,v>0 



with 



A 



E 

b,v>0 



-07 



("7" 



Ab,v 



A 



b.v 



B 



B, 



-67" 



(7- 



Vi\ ■ ■ ■Vb\ 



1=1 



E 

b,v>Q 
—07 



N 



N 



1_ 

N 



M-b 



Now, following the estimates for the approximation of a Poisson distribution 
by a Binomial given in Lemma 7.4.6], we can bound A^^^ and B^ by a 
quantity of the form j^. The proof is then easily finished. 

□ 



Let us relate now the law of (ui, . . . , Um) with /i| 



o,7~ 
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Lemma 3.13 We have 
with 

C2{N,m) = AU^e^""- E{D{N-m,\{F^^,,n\Jj,-fo)) 
+2Bo — — (e^^ - 1) + 



iV - m + 1 ' ' 
+2t/ooe2^- ^E(\R^-ma\ | exp - m«| }) . 

Proof: Notice that, invoking relation (jH^ and Theorem 13.11 we get 

m 



a,7~ 



(fe) («) j=l 

Then, the results follows easily from Lemmas 13 . 1 1 1 and 13 . 1 2[ Lemma 7.3.3 in 
and the triangular inequahty. 

□ 

We are now ready to end the proof of the main result concerning the 
magnetization of the system. 

Proof of Theorem 13.51 First of all, notice that by symmetry we have 

£((0-1), . . . , {arn)) = C{{aN-m+l), • • • , {(^n))- 

Furthermore, thanks to (j49|) and (j46j) and Lemma f3.3( we can write 
D{N, M, m, 7o) = sup d {C{{a^), <7) 



7S70 



-7<7o "^V V (AvO- (AvO- J'^-- 



<supd^. (^( 



7<7o 

2m^Q;7o(l + 0:70) 

4ma7o / [^70) [^070) 



«,7 
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Then, Lemma (3. 131 implies 

D(iV,M,m,7o) < 4 f/oo e^^- E {D{N - m, \{F]^^mT\, l^m|,7o)) 

+2So (e^^- - 1) 

A/ — m + 1 

+2?7oo e^^°= (^\Rm-ma\ | J„^| exp - ma|| 

^ 2m^Lo(7o) ^ 12m3a7^exp(7o) 



It is readily checked, as we did in (j29p . that 



E(|Jm|^) < ^ ^ (7o + 7o) (q^7o + (a^7o)^) > 

E (I J™fl < ^""" (70 + 37o + 7o) (a"^7o + 3(am7o)2 + (am7o)^). 

Thus, using the fact that Rm < Y where Y ~ B{mM, -^), together with the 
trivial bound Rm < M, there exists a constant > 1 such that 

D{N,M,m,jo) < AUooe'''-E{D{N-m,\{FJ^^Mn\IJ,-fo)) 

/^om^[a7^exp(|7o) + Lo( 7o ) 
N 

Now we are able to prove, by induction over A^, that 



(53) 



nfAT M ^ ^ 2i^om3[a74exp(|7o) + Lo(7o)] A^ 
U{i\ , M,m,'yo) < j;^ -, lor all m < — . 

Indeed, in order to check the induction step from A^ — 1 to A^, notice that 
|(i^;^M)1 < M and that 

E(|J„n<25^^(m3a7o^). 

So, using also that 



p(|J™|>y) <^E(|J„n< 
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and by our induction hypothesis and we have 
n/ Ar N / ^0"^^ [q7o exp(|7o) + Lo(7o 



+4 f/oo e 



2C/o, 



'2KoE (I J„|=^) 7o'exp(|7o) + ^0(70)] 'i2m^a^t 



N — m 



+ 



< 



Kqui^ [a7^ exp(|7o) + Lo(7o 



+4 f/oo e 



2C/0, 



/ 50irom3a7g [2a 7o4exp(|7o) + Lo(7o)] , 32m 



+ 



iV2 



S 4 

«7o 



Ar2 



Finally, since M < N — m, the proof easily follows from hypothesis ()43|1 . 



□ 



4 Replica symmetric formula 

Now that the limiting law of the magnetization has been computed, we can 
try to evaluate the asymptotic behavior of the free energy of our system, 
namely 

log 1 ^ exp {-HN,Mi<^)) 



To this purpose, set 



G(7) = «log(5;^exp(-7)^E 



where 



Vp := / (^exp (^M(^^fi'i,AfO-i))^ d^aai^i) x . . . x d/i„,^(xp) 

and means integration with respect to the product measure v on {—1, 1}^ 
such that J (Jidv = Xi. Then, the main result of this part states that: 
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Theorem 4.1 Set F such that F'{-f) = G{j) and F(0) = log 2 - au{0). 

Then, 7 < 7o and and hold true, we have 

biv(7)-i^(7)l<^, 
where K does not depend on 7 and N . 

Since PAr(O) = log 2 — au{0), the proof of the theorem is a consequence of the 
following proposition. 

Proposition 4.2 //7 < 70 and and hold, we have 

b^(7)-G(7)|<f, 
where p'j^{'y) is the right derivative ofp^il)- 

Proof: We divide the proof into two steps. 
Step 1: We will check that 

\Pn{i)-G\i)\<^. (54) 

where ^^(7) is defined as 



aE 



log / exp j n ^ gi,Mli,M^i + 9n,m^n^ ~ " ( XI 

\ \ i<N i<N 



Following the method used in Lemma 7.4.11 in we introduce the 
Hamiltonians 

-Hh,Mi<^) =^^[ ^9i,k {li,k + 5i,fc) o-i j , 

k<M \i<N J 

-HN,Mi<^) = XI ^ I ^3i,k min(l, (7^,^ + cr, j , 

k<M \i<N J 

where {Si±}i<i<N,i<k<M is a family of i.i.d. random variables with P{6i^k = 
1) = jj:, P{5i,k = 0) = 1 — We also assume that this sequence is indepen- 
dent of all the random sequences previously introduced. Observe that the 
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random variables min(l, {'yi,k + ^i,k)) ^-re i.i.d with Bernoulli law of parameter 
^, where 7' = 7 + 5 — ^. Set now, for j — 1, 2, 



log exp(-i^^,MW) 



Obviously, plf{S) — Pn{i'), and our first task will be to show that p\[{5) — 
p%{S) is of order 5'^: notice that 

PNi^) -pUS) = [log(exp (-i/^,MW + HlU<^)))2] , 

where (■)2 denotes the average for the Gibbs' measure defined by the Hamilto- 
nian H'^^m- Consider now Y^^^ = Ei,fe7i,fc(^i,fe- Since, -fi,k+5i,k = min(l, 7^,^+ 
Si,k) + li,kSi,k, on the set {Y^ j^ = 0}, we have Hlf j^^ = Hjf j^. So, we can 
write 



Pn{S)-pUS) = ttE l{y^^=i}log(exp(-i/^j^((T)+i/^_^(a))), 



N 
N 



l{y^^>2} log(exp {-H]^ j^{a) + HI j^{(t)))2 



Using that 



P(1V,M>2) = 1-(1-^ 



NM 



7V2 



NM-l 



7^ 



ATM ( 1 - ^ ) ^ < a25'72 



7V2 



and 



r \ NM-l r 

P{Y^^M = 1) = ™ ( 1 - ^ ) ^ < a7<5. 



Ar2 



Ar2 



it is easily checked that 



phiS)-p%{5) ^ K 
5— »o+ 5 ~ N' 



(55) 



which means that we can evaluate the difference p\[{5) — Pn{i) instead of 
P%i^) -Pn{i). 

However, following the same arguments as above, we can write 



Pn{^) -Pn{i) = -^E [log(exp (-I/^^mI'^) + ^JV,m(<t)))] 
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We consider now Yj^^m = J2ik^i,k- Notice that on the set {Yn^m = 0}, 
Hn,m = j^. So, we can write 



Pn{5) -Pn{i) 



N 



E [l 



{ViV,M=l} 



log(exp {-Hl^,y,j{a) + ifjv.Af (a 



[l{y^,M>2} log(exp {-Hl^^^ia) + i7^,M(a)))] 
Vi{6) + V2{6). 



Let us bound now Vi{S) and V2(5): since 



PiYN,M > 2) = 1 - 1 



we have 



6_ 

N 



NM 



NM 1 



NM-l 



N 



< a{NM-l)S^ 



\V2i5) \ < 2a^{NM -l)U^5'^. 
On the other hand, using a symmetry argument, we get 



Vi{5) = NM 1 - 



X E 



6_ 

N 



NM-l 



_5_ 



log / exp I ( gi,M'li,M(Ti + gN,M(TN^ ~ " ( X] dhAHhM'^^ 
\ \ i<N i<N 



Hence, we obtain that 

p]vW-m(7) ^ j.^ Vi{5) + V2{5) 

s — ^0+ 6 s — ^0+ 6 



aE 



log / exp ( M ^ gi,MliM(^i + 



i<N 



i<N 



Eventually, since 



y — >r 



PnH) -Pn{i) 



lim 



Pn{^) -Pn{i) 



7' - 7 0+ 5(1-^) 

putting together and (jSHl), we obtain 
Step 2: Let us check now that 

|G(7)-G^(7)l<f. 



(56) 



(57) 
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To this purpose, set 



^ := / exp ( u(^^gi^Mli,M(^i + gN,M(^N^ - u(^'^gi,M7i,M'^i) I 

\ \ i<N i<N J 

and let us try to evaluate first E[\&]: notice that 

^ _ EtTgSjv {'^iT.iKN 9i,Ani,M^i + 9n,m^n)) exp {-Hn,m-i{(^)) 

_ (exp {u{Y,i<N 9i,Mli,M'^i + 9n,m'^n)))m-i 

{exp[u{Y,i<N9i,M^i,M'^i)))M-l 

where {■)m-i denotes the usual average using the Hamiltonian Hn^m-i- Set 
Bp ■= {Z]j=i^7i,M = P,lN,M = 0} and B := {'Jnm = 1}, and let us denote 
by Em the conditional expectation given {7i,M, 1 < ^ < N}. Then 



E [^] = E 



N-l 



J2 i^p^Af m 

.P=0 



N-l 



p=0 



N 



+ E [IbEm m] 

{exp{Vp+i))M-i 



(58) 



N 



{exp{Vp))M-i 



^ N 



where 



Vp:=ul ^QLMCTi I 

\i<p J 



Set Xp = . . . , (cTp)). Then, using the triangular inequality and follow- 

ing the same arguments as in Proposition I2.8( we get, for a strictly positive 
constant K, 



E 



(exp(V"p+i))A/- 



{exp{Vp))M- 



E 



{exp{Vp+i))x^+^ 
{exp{Vp))x^ 



E 



{ew{Vp+i))M-i{exp{Vp))x^ - {exp{Vp+i))x,,+A^w{Vp)) 



Af-l 



{ew{yp))M-i{exp{Vp))xp 
< e'^'-E [|(exp(V;,+i))M-i - (exp(V;,+i))x,+J 
+ \{exp{Vp))M-i - {exp{Vp))xj,\] 



< e 



3Uc 



) j 

N ' 



(59) 
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Consider now some i.i.d. random variables zi, . . . ,Zp of law jj,a,-y such that 
(jH)) holds. Set Yp = {zi, . . . , Zp). Then, following the same arguments as 
above, we get, for a strictly positive constant K, 



E 



E 



(exp(l^p+i))y^^, 
(exp(V;))y^ 
< e^^-E [\{exp{Vp+,))x,^, - {exp{Vp+,))y^_^, 



{exp{Vp+i))x^+, 
{exp{Vp))x^ 



+ \{exp{Vp))x, - {exp{Vp))Y,\] 



< e 



3Uc 



'J j 

N ' 



(60) 



where in the last inequality we have used 

d 



and the fact that 



dx. 



■{exp{Vp))., < e' 



Notice that if is a random variable with law Bin(A^ — 1,;^), then 
E(iy^) < K, where K does not depend on N. So, putting together (jSHl), 
(|Sni) and (jnni), we get 



N-l 
p=0 



A^- 1 
P 



nJ 



N-p 



E 



(exp(V;+i))i 



p+i 



{exp{Vp))Y, 



+ 



K 
N' 



Using now similar arguments to those ones used in the proof of Lemma 13.111 
we get 



E[vl/]=^exp(-7)^E 



p=0 



{exp{Vp+i))Y^+, 
{exp{Vp))Y^ 



+ 



K 
N' 



(61) 



Eventually, once has been obtained, ()57|) can be established following 
the method used in Proposition 7.4.10 in jH], the remaining details being left 
to the reader. □ 



References 

[1] Bardina, X.; Marquez-Carreras, D.; Rovira, C; Tindel, S.: The p-spin 
interaction model with external field. Potential Analysis 21 no. 4 (2004) 
311-362. 



38 



[2] Cont, R.; Lowe, M.: Social distance, heterogeneity and social interactions. 
Preprint. 

[3] Pranz, S.; Toninelli, F.: The Kac limit for diluted spin glasses. Internat. 
J. Modern Phys. B 18 (2004), no. 4-5, 675-679. 

[4] Guerra, F.; Toninelli, F.: The thermodynamic limit in mean field spin 
glass models. Comm. Math. Phys. 230 (2002), no. 1, 71-79. 

[5] Hertz, J.; Krogh, A.; Palmer, R.: Introduction to the Theory of Neural 
Computation. Addison- Wesley Publishing Company, 1991. 

[6] Marquez-Carreras, D.; Rovira, C; Tindcl, S.: Assymptotic behavior of 
the magnetization for the perceptron model. To appear at Ann. Inst. H. 
Poincare Probab. Statist. 

[7] Mezard, M.; Parisi, G.; Virasoro, M.A.: Spin glass theory and beyond. 
World Scientific, 1987. 

[8] Shcherbina, M.; Tirozzi, B.: Rigorous solution of the Gardner problem. 
Comm. Math. Phys. 234 (2003), no. 3, 383-422. 

[9] Talagrand, M.: Spin Glasses: a Challenge for Mathematicians. Springer, 
2003. 

[10] Talagrand, M.: The Parisi solution; To appear at Annals of Math. 



39 



